CRPE 


REINVENTING 
PUBLIC EDUCATION 


How Much Have Students Missed Academically Because 
of the Pandemic? A Review of the Evidence to Date 


Panel of Experts 


Co-Chair Martin R. West, Harvard University 
Co-Chair Robin Lake, Center on Reinventing Public Education 


Julian Betts, University of California, San Diego 
Sarah Cohodes, Columbia University 
Brian Gill, Mathematica 
Andrew Ho, Harvard Graduate School of Education 
Susanna Loeb, Annenberg Institute at Brown University 
Billy McRae, Zearn 
Heather Schwartz, RAND 
James Soland, University of Virginia 
Michael Walker, Educational Testing Service 


July 2021 


HOW MUCH HAVE STUDENTS MISSED ACADEMICALLY BECAUSE OF THE PANDEMIC? 


| TABLE 


About this Series 


Overview 


Overview 


What We Know 
What We Don’t Know 
What We Need to Know 


Implications for Post-Pandemic Recovery 


OF CONTENTS 


OF PUPS cicisscecssccesescstecse tac stahaceceecccetconssbucscessevvhnutsaniaatetacunlatbncituecbubctdasslasba 


Appendix A. Characteristics of the Studies Included in this Review ..................... 


CENTER ON REINVENTING PUBLIC EDUCATION 


S 


HOW MUCH HAVE STUDENTS MISSED ACADEMICALLY BECAUSE OF THE PANDEMIC? 


About this Series 


This report is part of a series that aims to provide a definitive account of the best 
available evidence on how the COVID-19 pandemic has affected America’s students. 
The series is part of a broader effort, the Evidence Project, which brings together 
researchers and policymakers to advance solutions-oriented analysis of the K-12 
response to the COVID-19 pandemic. 


The Center on Reinventing Public Education compiled hundreds of studies and 
convened panels of experts to interpret what the data show. Three initial reports 
assess what we know to date about the pandemic’s effects on students’ academic 
progress, its effects on their mental health and social-emotional well-being, and 
its impact on students with disabilities. We will update these assessments and add 
more topics over the coming year as new data become available. 


We aim to present a coherent baseline of what we know, don’t know, and need to know 
at this stage of the pandemic. These reports are designed to help system leaders, 
community leaders, policymakers, researchers, philanthropies, the media, and others 
to define ambitious goals and clear metrics that ensure our education system meets 
every student’s needs over the coming years. 


The series of papers will culminate in the release of CRPE’s first annual Profile of the 
American Student. The report will provide a rigorous and nuanced assessment of 1) 
how extensive student needs and inequities are across a variety of dimensions, 2) 
how student needs vary across different dimensions and what that implies for policy 
and practice, and 3) what promising solutions and innovations are emerging to meet 
students’ needs. 


In future years, these reports will track progress toward repaying every student the 
educational opportunities they are owed after this traumatic and disruptive period. 
Our goal is to provide an ongoing assessment of student needs and a look forward 
toward restitution and recovery. 
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|. Overview 


The COVID-19 pandemic has disrupted every aspect of U.S. society over the past 18 months, 
but perhaps none more fundamentally than its K-12 schools. Virtually every school nationwide 
closed its doors in March 2020 for the remainder of the 2019-2020 academic year, replacing 
in-person instruction with hastily constructed online programming. Only a small fraction of 
U.S. students returned to fully operational schools last fall. The remainder experienced either 
partially or fully remote instruction for much of 2020-2021. 


Determining how the pandemic and associated disruptions to schooling have affected the 
development of students’ foundational academic skills is critical. Measures of literacy and 
numeracy while students are enrolled in K-12 schools are strong predictors of their future 
educational and labor-market success. Studies of past disruptions to schooling—for example, 
from prolonged teacher strikes—reveal adverse effects on students’ educational attainment 
and economic outcomes as adults.’ The economic and social consequences of the pandemic— 
employment loss, stress due to sick or dying relatives, the need to assume additional family 
responsibilities—may also have affected students’ academic progress independent of disruptions 
to schooling. Designing effective strategies to support students who have experienced 
setbacks brought on by the pandemic will require that educators and policymakers have a clear 
understanding of where students are. 


Yet gauging the academic impacts of the pandemic is hard: its disruptions to schooling also 
disrupted the assessment systems used to monitor students’ progress. Annual state tests were 
cancelled in spring 2020 in all states. Many school districts stopped administering interim 
assessments that are routinely used to determine whether students have mastered grade-level 
standards. Even where interim assessments continued, many students did not participate— 
and those who did not participate are likely to be the same students who experienced the 
greatest impact from the pandemic. Finally, changes in the mode of administration (e@.g., more 
students taking tests from their homes) raise questions about the validity of the data and their 
comparability to prior years. 


Despite these challenges, a body of evidence is emerging that sheds partial light on what has 
happened to student achievement overall and for demographic subgroups. In June 2021, the 
Center on Reinventing Public Education (CRPE) gathered a panel of 10 experts in education 
policy and assessment to review the available evidence on the pandemic’s impact on students’ 
academic skills. In this report, we summarize this evidence on how the pandemic and the 
converging social events of 2020 and 2021 affected K-12 students’ academic progress. Given 
important limitations of the evidence currently available, we also highlight what we don’t yet 
know; suggest what journalists, educators, and policymakers should be looking for as additional 
sources of data emerge; and offer an agenda that these experts feel should drive research in 
coming years. 


1 For a review of the evidence on the educational and economic consequences of lost learning and past school closures, see Eric A. 
Hanushek and Ludger Woessmann, The Economic Impacts of Learning Losses (Washington, DC: OECD, 2020). 
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Panel of Experts 


Co-Chair Martin R. West, Academic Dean and Professor of Education, Harvard University 


Julian Betts, Professor, Department of Economics, University of California, San Diego 
Sarah Cohodes, Associate Professor of Economics and Education at Teachers College, Columbia University 


Co-Chair Robin Lake, Director, Center on Reinventing Public Education 
! Brian Gill, Senior Fellow, Mathematica 


Andrew Ho, Professor of Education at the Harvard Graduate School of Education 
Susanna Loeb, Director, Annenberg Institute at Brown University 
i Billy McRae, Director of Research and Strategy, Zearn 
Heather Schwartz, Director, Pre-K to 12 Educational Systems Program; Senior Policy Researcher, RAND 
' James Soland, Assistant Professor of Quantitative Methods, University of Virginia 


Michael Walker, Distinguished Presidential Appointee, Educational Testing Service 


The panel reviewed studies that were gleaned from CRPE’s COVID-19 K-12 research database 
of more than 300 studies on the pandemic’s academic impacts. In particular, we sought to 
identify all studies that present data from standardized assessments of student achievement 
meeting three criteria: 


* The assessments were administered before and after the onset of the pandemic in March 
2020. 

¢« The assessments are designed to be comparable over time (Setting aside potential concerns 
about changes in administration conditions). 

e The assessments cover either a representative sample or a large share of the students 
enrolled in a school system (e.g., state or school district(s) in the U.S. 


We limited our analysis to studies meeting these standards in order to ensure some level of 
comparability before and during the pandemic and to make sure our inferences were based on 
broad populations, rather than a small number of students in potentially unique circumstances. 
We identified 12 studies that met our criteria. We provide details about these studies in the 
appendix. 


The studies we reviewed relied on the achievement measures of state test scores (in one case) 
or interim assessments many school districts use to track student progress. The post-pandemic 
data were gathered in fall 2020 or winter 2021. We did not include studies of course grades 
and other more subjective indicators of academic progress in our review because many school 
systems modified their grading practices amid the pandemic. We nonetheless draw on the 
results of those studies to inform our interpretation of the evidence included in the review. 
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We emphasize that standardized test scores are only partial indicators of students’ educational 
progress. Separate CRPE panels are examining the pandemic’s effects on social-emotional 
development and well-being, and on students with disabilities. CRPE and the expert panels 
plan to update these papers and add others in coming months and years as new studies are 
published. The studies we reviewed were generally not peer-reviewed but rather were shared 
by researchers as white papers in an effort to provide timely information. In the aggregate, 
however, the consistency of findings across studies paints a clear picture. 


Il. Overview of Findings 
What We Know 


1. Test scores show that the average student mastered less academic content because of 
the pandemic and associated disruptions to schooling. 


* On average, elementary and middle school students tested in fall 2020 performed 
at lower levels than (i) prior grade-level cohorts, and (ii) predictions based on pre- 
pandemic growth data. This does not literally mean that students know less than they 
did before the pandemic but does imply slower than typical progress. 


* On average, students tested in winter 2021 showed larger gaps relative to predictions 
based on pre-pandemic growth data than did students tested in fall 2020, implying 
that slower than typical progress continued well into the 2020-21 school year. 


* The precise magnitude of that slowdown is difficult to estimate nationally, due to 
increased rates of non-participation in testing and varying methods and units of 
measurement across studies. However, the weight of the evidence suggests students 
tested in the 2020-21 school year lagged pre-pandemic expectations by an amount 
roughly equivalent to several months of learning in a typical year. 


2. Other measures of instructional time, engagement, and academic progress strengthen 
the credibility of these findings and demonstrate further reason for concern. 


¢ Students had far less in-person instruction than usual over the past 18 months—in some 
cases, none at all. Remote instruction did not always include “live,” or “synchronous,” 
student-teacher interactions. Teachers report they covered much less material than 
usual. The extent of reduced learning time related to where students lived, local health 
conditions, and the politics of the area. 


¢« Many school districts reported sharp increases in course failure rates, especially for 
students from low-income households and students of color—in some cases despite 
explicit reductions in grading standards. 
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* Overall enrollment in public schools dropped by 3 percent nationally and by 13 percent 
among preschool and kindergarten students. Among students who remained enrolled 
in school, reported rates of absenteeism increased, and data from one state show a 
substantial increase in the number of students reported chronically absent (absent for 
more than one out of every three days).? 


3. The pandemic’s average impacts on academic achievement, while significant, mask 
substantial variation in impacts across subjects, grades, demographic groups, and 
geography. 


« |mpacts were generally greater in math than in reading, consistent with evidence 
indicating that schools play a larger role in the development of students’ math skills. 
Even so, the limited available evidence on oral reading fluency and literacy skills in the 
early elementary grades suggests substantial negative impacts in this domain. 


¢ |mpacts are consistently greater for economically disadvantaged students than those 
who are not. 


¢« |mpacts are consistently greater impacts for Black and Hispanic students than for 
white students. Relative impacts for Asian-American students, when reported, vary 
across studies. 


e Impacts are substantial across all grade levels for which data is available (K-8). 
Although impacts in elementary school (K-5) appear to be larger than in middle school 
(6-8) when they are measured relative to the variation in grade-level achievement, this 
apparent difference is less clear when measured relative to typical rates of achievement 
growth. 


¢« |mpacts on learning varied geographically, with some school districts experiencing 
little or no effect. For example, a study of Ohio found that, despite a substantial drop 
in average third-grade reading scores statewide, 10 percent of districts experienced no 
decline. A second study of a set of California school districts documented an increase 
in the variation in achievement growth rates across districts. 


e Impacts are greater where students had less access to in-person instruction, according 
to the two studies that examine this question. In theory, this could reflect the relative 
effectiveness of remote instruction or the fact that communities relying more on 
remote instruction were hardest hit by the pandemic in other ways. However, studies 
have shown that districts with higher infection rates were actually more likely to offer 
in-person instruction. There is little doubt that, on average, more in-person instruction 
produced more learning. 


4. The evidence to date likely understates both the average academic impacts of the 
pandemic and the opportunity and achievement gaps it produced. 


* Even setting aside the cancellation of state tests in spring 2020, far fewer students 
participated in standardized assessments in the 2020-21 school year than is typically 
the case. 


2 Increased absenteeism may not be surprising given widespread fears of the pandemic, actual COVID cases, and other pandemic-related 
hardships, but it does provide a credible mechanism to explain slower-than-typical learning. 
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e Economically disadvantaged students and students of color—groups that were 
hardest hit by the pandemic’s health and economic consequences—were less likely 
to participate in tests in the 2020-21 school year compared to their more advantaged 
peers. 


* These patterns imply that the data currently available to researchers may not reflect 
the pandemic’s full impact, and that these data may be particularly misleading when it 
comes to economically disadvantaged students and students of color. 


What We Don’t Know 


The sparse data available to date provide only the barest insight into the scope and magnitude 
of academic impacts brought on by the pandemic and associated school closures. The studies 
included in our review are understandably opportunistic, relying on available state or local 
assessments that typically cover only a few subject areas and grade levels. In particular, we lack 
almost any systematic data on the achievement of high school students and on the progress of 
students in subject areas beyond math and reading. In the early elementary grades, we have 
some evidence based on measures of oral reading fluency, but nothing that speaks to students’ 
math skills. The pandemic’s true impact will continue to reveal itself over the next year or two as 
we accumulate more studies based on interim assessments, state math and reading tests, and 
end-of-course exams in other subjects. It will be critical to continue monitoring that evidence— 
watching for whether students who suffered as a result of the pandemic are catching up, and 
for possible compounding effects for those who are not. 


We also have limited evidence on differences in the pandemic’s impact across different 
subgroups of students. For example, none of the available studies reports data separately by 
gender, despite the possibility that disruptions to in-person instruction could affect boys and girls 
differently. We have limited information on the achievement of English language learners and 
students with disabilities—and none at all on other special populations, including foster children 
and children experiencing homelessness. While several studies look at students separately by 
race and economic disadvantage, none look at the intersection of these characteristics. 


More generally, it remains unclear whether the average impacts documented to date conceal 
wide variation across individual students, with some students continuing to thrive and others 
experiencing larger setbacks. The available studies generally do not report whether the variation 
in achievement has increased in assessments conducted after the pandemic’s outbreak. 


Finally, it’s notable that we cannot really say at this point precisely how much learning the typical 
U.S. student missed—even in the grades and subjects where the most students have been 
assessed. The available studies that are national in scope are not necessarily representative 
in their coverage; others examine patterns in specific geographies. Despite efforts by some 
studies to examine the validity of tests given outside of school settings, uncertainty remains 
about whether they are truly comparable to tests given before the pandemic. Perhaps the most 
fundamental challenge is uneven test participation at the student- and school-level. Current 
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studies can and do observe the characteristics of students not participating in assessments, 
including in some cases prior achievement levels. However, they cannot rule out the possibility 
that students not participating have suffered more from the pandemic and become more 
disengaged from their schooling than those who do. 


What We Need to Know 


At the top of the priority list for researchers going forward should be understanding how 
students who have become most disengaged from schooling as a result of the pandemic fare 
over the next five to ten years. Addressing the needs of students who are not included in any 
of the evidence to date may be the most urgent challenge facing educators and policymakers. 
Increasing the coverage of assessments in the 2021-22 school year is therefore critical, as is 
tracking the progress of individual students, be it through diagnostic assessment, state exams, 
IEP meetings, etc. A useful first step toward this goal would be an “educational census” where 
the first and foremost question is: Who is here and who are we missing? 


A second key question is why more in-person instruction is associated with more positive 
learning outcomes—a pattern of evidence in both of the studies that examine this question. 
Is this due primarily to the mode of instruction? Or does it reflect other factors? The current 
evidence is likely sufficient to justify prioritizing a return to full-time in-person instruction for 
all students in the coming school year. Even so, better evidence on this question has clear 
implications for instructional design going forward. 


While the pandemic created crises for many students, it presented opportunities for others. 
Surveys and media reports show some students were able to manage well during remote 
learning and social isolation, and even thrive. Understanding which family supports, personal 
attributes, and educational circumstances contributed to that success will helo ensure any 
future disruptions in learning have less deleterious effects on students. Those clues to success 
may even help inform how schools can work better for all students. 


Equally critical is understanding whether the lags in achievement documented for most students 
are transitory, persistent, or—in the worst case scenario—compounding. Compounding effects 
are possible given the cumulative nature of academic development, in which gaps in previously 
gained skills can affect the rate of achievement growth going forward. It is even possible that 
some students have lost the ability to “do school” and will continue to experience slower than 
typical growth for an extended period if they receive conventional instruction. Yet it is also likely 
that many students have been learning in other ways that are not reflected in assessments, and 
these students’ academic skills may recover quickly as schools return to normal. Our lack of 
prior experience with disruptions to schooling at the scale of those produced by the pandemic 
makes future trajectories difficult to predict. Ongoing monitoring is therefore essential. 


Of course, the future achievement growth of students impacted by the pandemic will hinge 
on the success of educators’ work going forward. Whatever learning time has been lost can 
likely be recovered for most children, but only with intensive effort, strategic use of time, and 
evidence-based intervention. This implies an urgent need to document and study the success 
of various approaches. 
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Ill. Implications for Post-Pandemic Recovery 


The body of evidence to date leaves little doubt that the average student is academically behind 
past peers as a result of the pandemic, the ensuing school closures, and the move to remote 
learning. This statement is not an indictment of teachers, parents, or any others who tried to 
make the best of a chaotic situation, but the situation must be addressed quickly. Educators and 
policymakers should be especially focused on ensuring students have intensive support, as soon 
as possible, in math and early literacy. Studies show significant drops in average achievement in 
these areas. Leaving them unaddressed could cause many students long-term academic harm. 


What is most cause for concern, however, is the highly varied nature of results and the large 
numbers of students who appear to have fared very poorly. The studies we reviewed reveal 
that a large portion (though likely not the majority) of U.S. students seems to have learned very 
little in core subjects this year, failed classes at high rates, and been absent or missing from their 
schools. We are only beginning to understand the scope of the problems and the long-term 
consequences for those students, but their situations should compel urgent action to find those 
students and help them re-engage productively in school and in learning. 


Also troubling is the fact that the students who appeared to suffer the most academically were 
often those who were already not sufficiently supported in the prior educational environment: 
students of color and students from lower-income households. Critical action is required for 
these students. 


While the available studies do not help us understand the precise impact on students with 
disabilities, English language learners, and other vulnerable groups, these students require 
attention and data-informed intervention. Determining which students experienced the most 
adverse academic impacts may be a particularly good use of federal COVID-relief funding so 
that additional resources can be allocated to addressing their needs. 


Given the general dearth of data, it should be incumbent on school districts and states to be 
transparent about how they are measuring academic impacts, how they know students are or are 
not catching up, and how they are assessing and acting on the efficacy of various interventions. 
In the coming months and years, state exam results and widely used interim assessments will 
be important to watch but difficult to interpret given the holes in data and inconsistencies in 
test administration. CRPE and this panel will continue to update these findings and will provide 
guidance to reporters and policymakers on interpreting the latest results and body of research. 


The answer then, to the question, “How much have students missed academically as a result 
of the pandemic?” is only beginning to be understood but is clearly different for each student. 
Diverse impacts require diverse solutions. Returning to the status quo without addressing 
lost learning will do little to ameliorate learning loss. The answer to the question, “How much 
will they suffer long term?” depends on adult action. This baseline research review provides 
a roadmap for educators and policymakers to start from and will be critical to assess for the 
coming years. What we do from here matters. 
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Appendix A. Characteristics of the Studies 
Included in this Review 


As noted in the report text, we sought to include in this review all studies presenting data from 
standardized assessments of student achievement that met three criteria: 


* The assessments were administered before and after the onset of the pandemic in March 
2020. 


« The assessments are comparable over time (setting aside potential concerns about 
changes in administration conditions). 


¢ The assessments cover either a representative sample or a large share of the students 
enrolled in a school system (e.g., state or school district(s) in the U.S. 


The 12 studies we identified as meeting these criteria nonetheless varied along multiple 
dimensions, which are summarized in tables 1-4 below. 


Time Period: Eight of the twelve studies used assessments administered in fall 2021 (table 
1). These studies are therefore best understood as capturing the impact of the pandemic and 
resulting school closures in spring 2020 as of the start of the 2020-21 school year. Four studies 
used data gathered in the winter of that school year, typically just before or after the end of the 
calendar year, and therefore additionally capture the impact of pandemic-induced alterations in 
school practices in fall 2020. None of the studies reviewed speaks to the pandemic’s full impact 
at the end of the 2020-21 school year. 


Geography: The geographies focused on in each study ranged widely. Two focused on a single 
school district (Washington, D.C., and the School District of Philadelphia). Two focused on a set 
of school districts within the same state (California and Georgia). One covered the entire state 
of Ohio. The remainder are more national in scope, in some cases including data on students in 
all 50 states and the District of Columbia. None of these nationwide studies is truly nationally 
representative, however, as each relies on a sample of schools that participate in a particular 
interim assessment program (table 1). 


Subjects: The studies draw primarily on interim assessments of reading and math skills and 
therefore do not provide information on the pandemic’s effects on achievement in other subject 
areas. The study of Ohio focuses exclusively on English Language Arts, taking advantage of 
the fact that the third-grade state test was administered in that subject as scheduled in fall 
2020. One study of early elementary students focuses on Oral Reading Fluency, an important 
precursor skill in the reading domain (table 1). 


Grades: The studies as a set focus almost exclusively on students in elementary and middle 
school, with only a single study (of Philadelphia) providing any information on the achievement 
of high school students. Among younger students, more studies provide information on 
students in grades 3-8, as some studies excluded students tested in earlier grades due to 
concerns about the comparability of remote and in-school assessments in those grades (table 
1). 
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Sample Definition: Each of the studies sought to include as many students as possible while 
also attempting to ensure that the students tested after the pandemic were comparable to 
those tested before the pandemic. Some studies accomplished this at the level of the individual 
student, taking advantage of longitudinal data to include only those students tested both before 
and after the pandemic. The two studies by Curriculum Associates defined their sample based 
on schools administering their diagnostic assessments both before and after the pandemic but 
established criteria for the percentage of students participating in testing at each point in time. 
Finally, the studies of Ohio, Philadelphia, and Washington, D.C., included all students who were 
tested at either point in time (table 2). 


Test Mode: Only the study of the state of Ohio was able to rely on tests that were administered 
entirely in schools; all of the others had to rely on a mix of tests administered in schools and 
at home. Some studies simply combined results across modes, in some cases despite signs 
that students were performing better or soending more time on the tests when they were 
administered remotely. Other studies were forced to do so because of a lack of data on test 
mode for individual students. In some cases, researchers conducted validity studies designed 
to shed light on the comparability of in-school and remote testing to inform their analytic 
decisions. For example, NWEA found that students tested in grades K-2 (but not in higher 
grades) performed better than expected when tested at home, and therefore excluded those 
grades from their analysis (table 2). 


Method: Each of the studies reviewed sought to estimate the pandemic’s effect on students’ 
achievement relative to what they would have achieved had the pandemic not occurred, 
but they took various approaches to accomplishing this goal. Some simply compared the 
achievement levels of students tested in the 2020-21 school year to the achievement levels of 
prior cohorts of students tested in earlier years. For example, the study of Ohio third-graders 
compared the scores of students tested in fall 2021 to students tested in fall 2020. In making 
this comparison, the researchers at Ohio State University adjusted the 2021 test scores to 
account for demographic differences between students tested in each cohort. A larger number 
of studies used longitudinal data to compare individual students’ achievement in fall 2020 
or winter 2021 to predictions of where they would have been absent the pandemic based on 
historical growth data. Neither approach is able to provide information on the pandemic’s 
effects on students not participating in assessments in the 2020-21 school year (table 3). 


Reporting Metrics: The metrics used to report impacts on achievement also varied widely 
and included changes in the percentage of students meeting various achievement thresholds 
(e.g., at or above grade level, substantially below grade level); changes in national percentile 
ranks; percentage of expected growth; and standard deviations (i.e., relative to the variation in 
student achievement levels). Several studies converted estimates of achievement impacts into 
units of time (e.g., months or weeks of typical learning), though the methods for making these 
conversions also varied across studies (table 3). 


Subgroup Comparisons: In addition to reporting estimates of average achievement impacts 
for all students, most of the studies produced estimates of impacts on specific student groups. 
The most common breakdowns were by economic disadvantage (11 studies) and race/ethnicity 
(9 studies), though in some cases where researchers lacked individual data these subgroups 
were defined based on aggregate school characteristics. Only four of the twelve studies 
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produced separate estimates for students with disabilities and English language learners, while 
(surprisingly) none looked separately by student gender. Only two studies, those of Ohio and of 
three metro-Atlanta school districts, were able to relate estimates of the pandemic’s achievement 
impacts to measures of the amount of in-person (vs. remote) instruction students received. 
Both of these studies found that impacts were greatest where less in-person instruction was 
available (table 4). 


Table 1. Geography, Grades, and Subjects Covered by Reviewed Studies 


Time Organization (with Geography Grades Subject 
Period link to study) 
Fall2020 Renaissance Nationwide (50 states and | Ist-8th Reading/Math 
DC) 
Curriculum Associates | 8 states (reading) & 12 Ist-8th Reading/Math 
states (math) 
NWEA Nationwide 3rd-8th Reading/Math 
PACE 111 school districts in 22 Ist-4th Oral Reading 
states Fluency 
The Ohio State Statewide (Ohio) 3rd English Language 
University Arts 
PACE 18 California school Ath-8th Reading/Math 
districts 
Empowerk12 DCPS & DC public K-8th Reading/Math 
charters 
School District of School District of K-12th Reading/Math 
Philadelphia Philadelphia 
Winter Renaissance Nationwide (50 states and | Ist-8th Reading/Math 
2021 DC) 
Curriculum Associates | Nationwide (49 states and | Ist-8th Reading/Math 
DC) 
Georgia State 3 Metro-Atlanta school Ath-8th Reading/Math 
University districts 
PACE 19 California school Ath-8th Reading/Math 
districts 
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Table 2. Sample Definition and Mode of Test Administration of Reviewed 


Studies 

Organization (with Sample Test Mode 
link to study) 

Renaissance Students tested in fall 2019 and fall 2020. In-school and 


Curriculum Associates 


NWEA 


PACE 


The Ohio State 
University 


PACE 


Empowerk12 


School District of 
Philadelphia 


Inclusion criteria: (1) Enrolled in districts/ 
grades where at least 200 students 

tested in school in fall 2020; (2) Enrolled 

in districts/grades where at least 70% of 
students tested in fall 2019; (3) Enrolled in 
schools where number of students tested in 
fall 2020 was >50% of number of students 
tested in fall 2019. 


Levels: Schools that tested at least 10 
students in fall 2019 and fall 2020. Growth: 
Students tested in fall 2019, winter2020, and 
fall 2020. 


Only students tested before and after 
closures. 


Students tested in fall 2020 (81% vs. ~95% 
in a typical year); estimates adjusted for 
demographic differences. 


Students tested in fall 2019 and fall 2020. 


Students tested in fall 2020 and in fall 2019 
or winter 2020. 


Students tested in winter 2020 and fall 
2020. 


remote (combined 
despite performance 
differences) 


In-school only (self- 
reported remote test- 
takers excluded) 


In-school and remote 
(K-2 excluded after 
comparability study) 


Remote only 


In-school only 


In-school and remote 
(mode data not 
available) 


In-school and 
remote (students 
with atypical testing 
times/behavior 


In-school and remote 
(combined despite 
longer remote test 
time) 
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Organization (with 
link to study) 


Sample 


Test Mode 


Renaissance 


Curriculum Associates 


Georgia State 


University 


PACE 


Students tested in fall 2019, fall 2020, and 
winter 2021. 


Tested in winter 2020 and enrolled in school 
that tested 50-200% of the number of 
students tested in prior years. 


Tested in winter 2020 and enrolled in school 
that tested 50-200% of the number of 
students tested in prior years. 


Students tested in fall 2019 and winter 2021. 


In-school and remote 
(combined despite 
testing differences) 


In-school only (self- 
reported remote test- 
takers excluded) 


In-school only (self- 
reported remote test- 
takers excluded) 


In-school and remote 
(mode data not 
available) 
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Table 3. Methods and Primary Reporting Metrics Used in Reviewed 


Studies 


Organization (with 
link to study) 


Method 


Reporting metrics 


Renaissance 


Curriculum Associates 


NWEA 


PACE 


The Ohio State 
University 


PACE 


Empowerk12 


School District of 
Philadelphia 


Renaissance 


Curriculum Associates 


Georgia State 
University 


PACE 


Compare to predictions based on historical 
growth data. 


Compare levels to students in the same 
grades/grades in prior years. 


Levels: Compare to prior year. Growth: 
Compare to predictions based on historical 
growth data. 


Longitudinal growth model with data before 
and after closures. 


Compare 2020 achievement to 2019 
achievement with demographic 
adjustments. 


Compare to predictions based on historical 
growth data. 


Growth: Compare to national growth norms. 
Levels: Compare to prior years. 


Compare NPR in winter 2021 to fall 2020. 


Compare to predictions based on historical 
growth data. 


Compare winter 2020 achievement to 
average of prior years. 


Compare to predictions based on historical 
growth data. 


Compare to predictions based on historical 
growth data. 


Percentiles, weeks of 
learning 


% twot grade levels 
below 


Percentiles, scale 
scores, growth 


Growth in Oral 
Reading Fluency 
(words per minute) 


Standard deviations; 
months of learning 


Scale scores 


% of typical fall-to-fall 
growth; % on-track 
for proficiency; % 
two+ grade levels 
behind 


Changes in National 
Percentile Rank 


Student Growth 
Percentile, percentile 
ranks 


% of students on/ 
below grade level 


Months of learning 


Months of learning 
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Table 4. Subgroup Comparisons Included in Reviewed Studies 


Organization Economic Race/ Gender English In-person Students with 

disadvantage ethnicity language vs. Remote disabilities 
learners instruction 

Renaissance Y* Y N Y Y N 

Curriculum Y* YY N N N N 

Associates 

NWEA Y* Y N N N N 

PACE N N N N N N 

The Ohio State Y* Y N Y Y N 

University 

PACE Y N N Y N N 

Empower K12 Y Y N Y N N 

School District of Y Y N N N N 

Philadelphia 

Renaissance 0 Y N Y Y N 

Curriculum Y* Y* N N N N 

Associates 

Georgia State ar N N N N Y 

University 

PACE Y Y N Y Y N 


* Indicates that the subgroup comparison was defined based on aggregate school characteristics. 


AV CENTER ON REINVENTING PUBLIC EDUCATION 


18 


HOW MUCH HAVE STUDENTS MISSED ACADEMICALLY BECAUSE OF THE PANDEMIC? 


About the Center on Reinventing Public Education 


CRPE is a nonpartisan research and policy analysis center at the University of Washington 
Bothell. We develop, test, and support bold, evidence-based, systemwide solutions to address 
the most urgent problems in K-12 public education across the country. Our mission is to reinvent 
the education delivery model, in partnership with education leaders, to prepare all American 
students to solve tomorrow’s challenges. Since 1993 CRPE’s research, analysis, and insights 
have informed public debates and innovative policies that enable schools to thrive. Our work is 
supported by multiple foundations, contracts, and the U.S. Department of Education. 
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